According to Andrej Karpathy on Twitter, nanochat can now train large language models (LLMs) with GPT-2 level capabilities for less than $100, specifically around $73 in just over 3 hours on a single 8XH100 node. This represents a dramatic reduction in both time and cost compared to the original GPT-2 training by OpenAI in 2019, which required 32 TPU v3 chips running for seven days at a total cost of approximately $43,000. The advancement leverages optimizations such as Flash Attention 3 kernels, the Muon optimizer, and improved residual pathways. As reported by Karpathy, these developments not only make LLM prototyping significantly more accessible but also demonstrate a continued trend of rapidly decreasing training costs, opening new business opportunities for startups and researchers in the AI field.